Augmented reality robotic system visualization

ABSTRACT

A technique for displaying a representative path associated with a robotic device. The technique includes detecting at least one reference point within a first image of a workspace, generating the representative path based on path instructions associated with the robotic device and the at least one reference point, and displaying the representative path within the workspace.

BACKGROUND Field of the Embodiments

The various embodiments relate generally to human-machine interfacesand, more specifically, to augmented reality robotic systemvisualization.

Description of the Related Art

Recent advances in the fields of robotics and computer-aidedmanufacturing (CAM) have enabled robotic devices to perform complextasks traditionally performed by humans. In particular, improvements inthe dexterity and sensing capabilities of robotic devices have enabledthese types of devices to be adapted to perform a variety of tasks,often in a manner more efficient than their human counterparts. As aresult, robotic devices are becoming an important component in manyindustries.

Increasingly, tasks performed by robotic devices are performed in tandemwith humans and/or with other robotic devices. Because these types oftasks often require a robotic device to be in close proximity withhumans and/or other robotic devices, robot-human collisions and/orrobot-robot collisions are a significant risk factor. While the lattertype of collisions may be costly and require significant repairs to oneor both of the robots involved in the collision, the former type ofcollisions may result in great bodily harm or human death. Due to theextreme nature of these risks, many industries are reluctant tointegrate robotic devices into applicable operations and processes.

In order to reduce the risk of collisions, conventional softwareapplications (e.g., a CAM software application) allow a user to previewthe expected movements of a robotic device associated with a particularoperation on a computer screen. The user is then able to modify variousaspects of the operation in order to change the path of the roboticdevice to suit the specific requirements of the user.

One drawback of the above approach is that the preview generated by thesoftware application typically does not take into account other roboticdevices, humans, machinery, etc. that may be present in the workenvironment. Thus, when the user views the preview on his or hercomputer screen, he or she must then attempt to mentally visualize themovements and potential interactions between the robot being modeled inthe application software and other robotic devices, humans, machinery,etc. in the actual work environment as the robot being modeled executestasks in that work environment. Because such mental visualizations areincredibly difficult, this conventional process is highly error proneand inefficient. Consequently, users oftentimes program robotic devicesin an overly cautious and non-optimized manner to avoid all risk ofcollision, or users end up making mistakes that result in collisionsduring operation.

Further, because of safety concerns and regulations, if a user believesthat an operation may cause a robotic device to collide with an objectduring operation, then the user typically has to reprogram one or moreaspects of the modeled behavior of the robotic device. Such an approachcan become quite iterative, tedious, and time consuming, which canresult in further reluctance to integrate robotic devices intoapplicable operations and processes.

As the foregoing illustrates, improved techniques for visualizing andmodifying an operation to be performed by a robotic device would beuseful.

SUMMARY

Embodiments of the present disclosure set forth a method for displayinga representative path associated with a robotic device. The methodincludes detecting at least one reference point within a first image ofa workspace, generating the representative path based on pathinstructions associated with the robotic device and the at least onereference point, and displaying the representative path within theworkspace.

Further embodiments provide, among other things, a non-transitorycomputer-readable storage medium and a system configured to implementthe techniques set forth above.

At least one advantage of the techniques described herein is that athree-dimensional representation of the path of a robotic device can beviewed within the operating environment of the robotic device, allowingpotential collisions to be detected, before such collisions take place.Additionally, the path of the robotic device can be modified via userinput within the operating environment itself, and updates to the pathcan be immediately viewed, for example, via an augmented realitydisplay. Accordingly, robotic devices can be more safely and efficientlyimplemented in a variety of processes. Further, detailed informationabout robot components (e.g., joint angles) can be viewed, enabling auser to make more informed decisions when planning complicated movementsof a robot.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

So that the manner in which the recited features of the one or moreembodiments set forth above can be understood in detail, a moreparticular description of the one or more embodiments, brieflysummarized above, may be had by reference to certain specificembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlytypical embodiments and are therefore not to be considered limiting ofits scope in any manner, for the scope of the various embodimentssubsumes other embodiments as well.

FIG. 1 is a conceptual illustration of a computing device configured toimplement one or more aspects of the present invention;

FIG. 2 illustrates a perspective view of a tracker and a workpieceincluded in a work environment, according to various embodiments of thepresent invention;

FIG. 3 illustrates a virtual robotic device displayed within the workenvironment FIG. 2, according to various embodiments of the presentinvention;

FIGS. 4A-4C illustrate a path visualization technique implemented forthe graphical representation of the robotic device of FIG. 3, accordingto various embodiments of the present invention;

FIG. 5 is a flow diagram of method steps for displaying a virtual pathassociated with a robotic device, according to various embodiments ofthe present invention;

FIG. 6 illustrates a technique for modifying the path of the virtualrobotic device within the work environment of FIG. 3, according tovarious embodiments of the present invention; and

FIG. 7 is a flow diagram of method steps for modifying a virtual path ofa robotic device via user input, according to various embodiments of thepresent invention.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth toprovide a more thorough understanding of the embodiments of the presentdisclosure. However, it will be apparent to one of skill in the art thatthe embodiments of the present disclosure may be practiced without oneor more of these specific details.

FIG. 1 is a conceptual illustration of a computing device 100 configuredto implement one or more aspects of the present invention. As shown,computing device 100 includes, without limitation, a processor 102,input/output (I/O) devices 104, and a memory 110. Memory 110 includes apath visualization application 112 configured to interact with adatabase 114. In various embodiments, the computing device 100 may becoupled to a display device 120, a head-mounted display device 122,and/or a sensor 124.

The processor 102 may be any technically feasible form of processingdevice configured to process data and execute program code. Theprocessor 102 could be, for example, and without limitation, a centralprocessing unit (CPU), a graphics processing unit (GPU), anapplication-specific integrated circuit (ASIC), a digital signalprocessor (DSP), a field-programmable gate array (FPGA), and so forth.

Memory 110 may include a memory module or a collection of memorymodules. Database 114 within memory 110 may store images, video,algorithms, graphical representations, rendering engines, trackingsoftware, path instructions, lookup tables, and/or other types of dataassociated with tracking reference points, generating graphicalrepresentations, displaying graphical representations, etc. The pathvisualization application 112 within memory 110 is executed by theprocessor 102 to implement the overall functionality of the pathvisualization system 101. For example, and without limitation,multimedia content (e.g., images, video, etc.) received via the pathvisualization system 101 could be processed by the path visualizationapplication 112 to detect and track one or more reference points withina particular environment and generate a representation of the path of arobotic device based on the reference point(s). The path visualizationsystem 101 could then display the representation of the path of therobotic device to a user, such as by overlaying the representation ontoreal-time images (e.g., images acquired and displayed with typicalcapture and display latency) of the environment acquired via a sensor124 and displayed via a display device 120 (e.g., a tablet computer).Additionally, in some embodiments, the path visualization system 101could display the representation of the path of the robotic device tothe user by overlaying the representation onto the field of view of theuser via a head-mounted display device 122 (e.g., a transparentaugmented reality display).

I/O devices 104 may include input devices, output devices, and devicescapable of both receiving input and providing output. For example, andwithout limitation, I/O devices 104 could include wired and/or wirelesscommunication devices that send data to and/or receive data from asensor 124 (e.g., a camera, depth sensor, radar sensor, etc.), a displayscreen (e.g., display device 120, head-mounted display device 122, eta),a storage device, a networking device, and/or another computing device.

The display device 120 and the head-mounted display device 122 mayinclude any technically feasible device for displaying images of anenvironment and/or graphical representations of a robotic device, a pathof a robotic device, information associated with a robotic device, etc.In some embodiments, the display device 120 is included in a mobilecomputing device, such as a tablet computer. In addition, thehead-mounted display device 122 may be included in a mobile computingdevice, such as a virtual reality headset (e.g., Oculus Rift®) and/or anaugmented reality headset (e.g., Microsoft® HoloLens®).

The sensor(s) 124 may include, without limitation, visible lightsensors, thermal imaging sensors, laser based devices, ultrasonicsensors, infrared sensors, radar sensors, depth sensors, globalpositioning system (GPS) devices, magnetometers, inertial sensors,gyroscopes, accelerometers, etc. The sensor(s) 124 enable referencepoints (e.g., visual features, physical features, GPS coordinates, etc.)within the environment to be tracked and/or images of the environment tobe acquired. Although only one sensor 124 is illustrated in FIG. 1, invarious embodiments, multiple sensors 124 may be implemented to trackreferences points, acquire images, etc.

Generally, computing device 100 is configured to coordinate the overalloperation of the path visualization system 101. In other embodiments,the computing device 100 may be coupled to, but separate from othercomponents of the path visualization system 101. However, theembodiments disclosed herein contemplate any technically feasible systemconfigured to implement the functionality of the path visualizationsystem 101.

Computing device 100 as a whole may be a microprocessor, anapplication-specific integrated circuit (ASIC), a system-on-a-chip(SoC), a desktop computing device, a mobile computing device such as atablet computer or cell phone, a media player, and so forth. In someembodiments, computing device 100 is integrated with the display device120, head-mounted display device 122, and/or sensor 124. In otherembodiments, the computing device 100 may be coupled to, but separatefrom the display device 120, head-mounted display device 122, and/orsensor 124. In such embodiments, the display device 120, head-mounteddisplay device 122, and/or sensor 124 may include separate processorsthat receive data (e.g., images, instructions, etc.) from and transmitdata to the computing device 100. However, the embodiments disclosedherein contemplate any technically feasible system configured toimplement the functionality of the path visualization system 101.

FIG. 2 illustrates a perspective view 200 of a tracker 210 and aworkpiece 220 included in a work environment, according to variousembodiments of the present invention. In various embodiments, theperspective view 200 represents an image of an environment in which agraphical representation of a robotic device, a robotic device path,and/or other data related to the robot is to be displayed, such as byoverlaying the graphical representation(s) onto the image. In someembodiments, the perspective view 200 represents a field of view of auser in which a graphical representation is to be displayed, such as bydisplaying the graphical representation via a transparent displaypositioned in front of one or both eyes or the user.

As shown, a tracker 210 may be positioned within a particularenvironment to indicate the location at which a graphical representationis to be displayed. In general, the tracker 210 may include anyidentifiable shape, color, pattern, etc. that is capable of beingrecognized by via the sensor(s) 124. In various embodiments, the tracker210 may include a distinct shape and/or pattern that enables the pathvisualization application 112 to determine the location, orientation,and/or size of the tracker 210 within the perspective view. In suchembodiments, the path visualization application 112 is then able totranslate, rotate, and/or scale the graphical representation in order toproperly display the graphical representation within the perspectiveview.

Further, in some embodiments, instead of or in addition to using atracker 210, the path visualization application 112 may detect othertypes of reference points within an environment in order to translate,rotate, and/or scale the graphical representation so that the graphicalrepresentation is properly displayed within the perspective view. Forexample, the path visualization application 112 could detect one or moreedges and/or corners of the environment and/or one or more features ofstatic objects within the environment and use these reference points totrack changes to the perspective view 200 caused by moving the sensor(s)124, moving a display device 120, 122, etc. In a specific example, thepath visualization application 112 could detect the location of arobotic device associated with the graphical representation to bedisplayed and track changes to the perspective view based on changes tothe location, orientation, and/or size of the robotic device.

FIG. 3 illustrates a graphical representation of a robotic device 230displayed within the work environment of FIG. 2, according to variousembodiments of the present invention. As shown, the path visualizationapplication 112 generates and displays the graphical representation ofthe robotic device 230 based on the one or more reference points (e.g.,tracker 210), such as by translating, rotating, and/or scaling thegraphical representation based on the reference point(s). For example,the path visualization application 112 could generate a graphicalrepresentation of the robotic device 230 having the correct orientationand size for the perspective view and display the graphicalrepresentation of the robotic device 230 at a location proximate to thereference point(s).

Further, as described above, if the robotic device associated with thegraphical representation 230 is physically present in the environment,then the path visualization application 112 may track one or moreaspects of robotic device itself in order to properly generate anddisplay the graphical representation 230. For example, the pathvisualization application 112 could display the graphical representationof the robotic device 230 within the perspective view 200 atsubstantially the same location as the physical robotic device, enablingthe path visualization techniques described below to be performed withrespect to the actual location of the physical robotic device.

The path visualization application 112 may cause the graphicalrepresentation of the robotic device 230 to be displayed as an opaquegraphic, blocking content located behind the graphical representation230 from being viewed by the user, or as a translucent graphic, causingcontent located behind the graphical representation 230 to be at leastpartially visible. In some embodiments, the path visualizationapplication 112 may generate a translucent graphical representation ofthe robotic device 230 when the graphical representation 230 is beingdisplayed on top of (or proximate to) the physical robotic device sothat the graphical representation 230 appears as a “ghosted” version ofthe physical robot device.

For clarity of illustration and explanation, only one graphicalrepresentation of a robotic device 230 is shown in FIGS. 3-4C. However,in various embodiments, the path visualization application 112 maydisplay multiple graphical representations, each corresponding to adifferent robotic device. Additionally, the graphical representation(s)may be displayed proximate to one or more physical robotic devicesand/or humans, enabling a user to view how the robotic devices would bepositioned in a work environment while performing one or more tasks. Forexample, in some embodiments, the path visualization application 112could simultaneously generate and display two or more graphicalrepresentations of different robotic devices that are traversingdifferent paths that may or may not intersect. In such embodiments, theuser is able to view how the robotic devices would interact whenperforming task(s), such as whether the robotic devices are likely tocollide, whether the robotic devices are properly working in unison toperform a shared task (e.g., passing objects between one another),and/or whether the robot devices are likely to move proximate to a humanor collide with other objects in the environment.

FIGS. 4A-4C illustrate a path visualization technique implemented forthe graphical representation of the robotic device 230 of FIG. 3,according to various embodiments of the present invention. As describedabove, the path visualization system 101 enables a user to visualize apath associated with an operation to be performed by the robotic device.In some embodiments, path visualization includes displaying an animatedgraphical representation of the robotic device 230, an expected path ofone or more components of a robotic device (e.g., an end effector), atrail of a path traversed by a component of a robotic device or agraphical representation of a robotic device 230, and/or various typesof information associated with components of a robotic device. Forexample, during path visualization, the path visualization application112 could process path instructions (e.g., numerical control (NC)instructions, such as G-code instructions) associated with an operationto be performed by a robotic device to generate a graphicalrepresentation of a path 400. The path visualization application 112could then display the graphical representation of a path 400 via thedisplay device 120 or the head-mounted display device 122.

The graphical representation of the path 400 generated by the pathvisualization application 112 may include path segments 405 to betraversed by one or more robotic device components (e.g., path segmentsspecified by path instructions). For example, the graphicalrepresentation of the path 400 for a particular operation to beperformed by a robotic device could extend from a starting point of anend effector 232 of the robotic device to one or more locations 415 atwhich the end effector 232 would contact one or more workpieces 220.

The graphical representation of the path 400 generated by the pathvisualization application 112 may further include control points 410associated with one or more of the path segments 405. For example, asshown, a control point 405 may be positioned proximate to the end(s) ofpath segments 405 to indicate distinct movements to be performed by arobotic device. Additionally, as described in further detail inconjunction with FIGS. 6 and 7, one or more control points 410 may beinteractive, enabling a user to select and move controls points 410 withhis or her hand(s) in order to modify the path to be traversed by therobotic device.

In some embodiments, the graphical representation of the path 400includes an area or a volume that will be occupied by the robotic devicewhile performing a particular operation specified by a set of pathinstructions. In such embodiments, the user is able to simultaneouslyview all of the locations that will be occupied by the robotic devicefor a given time slice when performing the operation. For example, thepath visualization application 112 could generate and display shadedvolumetric boundaries that represent all of the locations of anoperating environment through which a component of the robotic devicewill travel during an operation. The user would then be able to walkaround the environment and determine whether any boundary of the shadedvolumetric boundaries intersects an object in the environment and/or alocation at which an object may be present during the operation of therobotic device. In another example, the path visualization application112 could generate and display volumetric boundary indicators thatrepresent all of the locations of an operating environment through whicha component of the robotic device will travel during an operation.

For clarity of illustration, the path segments 405 shown in FIGS. 4A-4Care illustrated as straight lines. However, in various embodiments, oneor more robotic device components may travel in non-linear paths whenperforming a particular operation. Accordingly, the graphicalrepresentation of a path 400 may include any type of path segment,including straight lines, curves, splines, continuous path segments,non-continuous path segments, etc. For example, when the graphicalrepresentation of the path 400 includes curves, splines, etc., the pathvisualization application 112 may also display control points 410 thatdefine the curvatures of path segments 405 included in the graphicalrepresentation of the path 400.

In some embodiments, the path visualization application 112 may furthergenerate and display one or more graphical indicators 420 that representvarious types of information associated with components of the roboticdevice. For example, as shown in FIG. 4A, a graphical indicator 420 maybe displayed proximate to one or more joints of the graphicalrepresentation of the robotic device 230 to indicate the angles of thejoints (e.g., angle X°, angle Y°, angle Z°) as the robotic deviceperforms an operation. In the same or other embodiments, other types ofinformation may be displayed proximate to one or more components of therobotic device, such as the speed of a particular component, theacceleration of a particular component, and/or the distance of aparticular component from another object or location. For example, thepath visualization application 112 could generate a graphical indicator420 that displays the speed and/or acceleration of the end effector 232and/or one or more joints of the robotic device when performing anoperation. In some embodiments, the path visualization application 112could dynamically update the information included in the graphicalindicators 420 as the visualization of the operation progresses. Inaddition, the path visualization application 112 could dynamicallyupdate the positions at which the graphical indicators 420 are displayedin the perspective view so that the graphical indicators 420 remainproximate to the corresponding components of the robotic device.

In some embodiments, the graphical indicators 420 include a joint limitmeter that indicates how close each joint is to its physical limit. Byproviding this type of information to a user, the user can make moreinformed decisions during a path-planning process. For example, if auser notices that a joint is approaching its physical limit—which couldprevent the robot from moving in a certain manner later in theprocess—then the user could change the path and/or configuration of therobot to avoid such issues downstream.

As shown in FIGS. 4B and 4C, as the visualization of an operationprogresses, the path visualization application 112 may animate thegraphical representation of the robotic device 230, enabling the user tovisualize how and where the robotic device would move during theoperation. In some embodiments, as a component of the graphicalrepresentation of the robotic device 230 moves along the graphicalrepresentation of a path 400, portions (e.g., path segments 405) of thegraphical representation of the path 400 that have already beentraversed may be removed by the path visualization application 112. Inother embodiments, the graphical representation of the path 400 isdisplayed as a trail, such as by displaying portions of the graphicalrepresentation of a path 400 as the corresponding component of thegraphical representation of the robotic device 230 traverses thoseportions.

In various embodiments, the path visualization application 112 trackschanges to the location, size, and/or orientation of the referencepoints during the visualization process. Accordingly, if the user movesthe display device 120, 122 and/or sensor(s) 124 during thevisualization process, then the path visualization application 112modifies the locations, sizes, and/or orientations of the graphicalrepresentation of the robotic device 230, the graphical representationof the path 400, and/or the graphical indicators 420 so that thesegraphics are properly displayed from the perspective of the user. Forexample, with respect to the graphical representation of the roboticdevice 230, the path visualization application 112 could modify thelocation, size, and orientation each time the perspective view isaltered so that the graphical representation of the robotic device 230accurately represents the location, size, and orientation of thephysical robotic device during a particular operation. Further, withrespect to the graphical representation of the path 400, the pathvisualization application 112 could modify the location, size, andorientation each time the perspective view is altered so that thegraphical representation of the path 400 accurately represents the paththat would be traveled by a component of the robotic device during aparticular operation.

FIG. 5 is a flow diagram of method steps for displaying a pathassociated with a robotic device, according to various embodiments ofthe present invention. Although the method steps are described inconjunction with the systems of FIGS. 1-4C, persons skilled in the artwill understand that any system configured to perform the method steps,in any order, falls within the scope of the various embodiments.

As shown, a method 500 begins at step 510, where the path visualizationapplication 112 processes path instructions (e.g., numerical control(NC) instructions, such as G-code instructions) associated with arobotic device to determine a path of one or more components of therobotic device. In some embodiments, the path visualization application112 processes path instructions associated with an operation to beperformed by the robotic device to determine, as a function of time, thecoordinates of one or more components (e.g., end effector 232, joints,arms, etc.) within a three-dimensional space.

At step 520, the path visualization application 112 acquires one or moreimages via the sensor(s) 124. Then, at step 530, the path visualizationapplication 112 detects the location(s) of one or more reference points(e.g., a tracker 210) included in the image(s). At step 540, the pathvisualization application 112 generates one or more graphics based onthe location(s) of the reference point(s) and displays the graphics viathe display device 120 or the head-mounted display device 122. Forexample, at step 540, the path visualization application 112 couldoverlay the graphic(s) onto an image acquired at step 520 and/or displaythe graphic(s) on a translucent display (e.g., head-mounted displaydevice 122).

In some embodiments, at step 540, the graphic(s) generated and displayedby the path visualization application 112 vary as a function of time,based on which portion of the path is currently being visualized. Forexample, as shown in FIG. 4A, if the beginning of the path (i.e., t=0)is being visualized, then the path visualization application 112 couldgenerate and display the entire graphical representation of the path 400and a graphical representation of the robotic device 230 in which thecomponents are configured in a starting position. By contrast, if asubsequent portion of the path is being visualized (e.g., as shown inFIG. 4B), then the path visualization application 112 could generate anddisplay a graphical representation of the robotic device 230 in whichthe components are configured in a different position.

Next, at step 550, the path visualization application 112 determineswhether additional path data should be visualized. For example, at step550, the path visualization application 112 could determine thatadditional portions of the path (e.g., the portions of the path shown inFIG. 4C) have not yet been generated and/or displayed. If the pathvisualization application 112 determines that additional path datashould be visualized, then the method 500 returns to step 520, where oneor more additional images are acquired. If the path visualizationapplication 112 determines that additional path data should not bevisualized, then the method 500 terminates.

FIG. 6 illustrates a technique for modifying the path of the roboticdevice within the work environment of FIG. 3, according to variousembodiments of the present invention. As described above, one or morecontrol points 410 associated with a path may be interactive, enabling auser to select and move the controls points 410 with his or her hand(s)in order to modify the path to be traversed by the robotic device. Forexample, as shown, a user could grab control point 410-1 and move thecontrol point 410-1 to his or her right, causing the path visualizationapplication 112 to move path segment 405-1 and path segment 405-2 to theright and modifying the location at which the end effector 232 willcontact the workpiece 220.

In other embodiments, when a user grabs control point 410-1 and movesthe control point 410-1 to his or her left or right, the pathvisualization application 112 modifies the locations, lengths,curvatures, and/or angles of the path segments 405-1, 405-2 and controlpoint 405-1, but does not modify the location at which the end effector232 will contact the workpiece 220. Accordingly, in such embodiments,the path of a component of the robotic device can be modified withoutsubstantially affecting the operation being performed on the workpiece.Additionally, in various embodiments, the user is able to grab andmodify one or more path segments 405 in order to modify the path of acomponent of the robotic device. Further, in some embodiments, the useris able to grab and modify components of the graphical representation ofthe robotic device 230 in order to modify the locations and/or angles ofthe components when performing an operation.

FIG. 7 is a flow diagram of method steps for modifying a path of arobotic device via user input, according to various embodiments of thepresent invention. Although the method steps are described inconjunction with the systems of FIGS. 1-6, persons skilled in the artwill understand that any system configured to perform the method steps,in any order, falls within the scope of the various embodiments.

As shown, a method 700 begins at step 710, where the path visualizationapplication 112 acquires one or more images via the sensor(s) 124. Atstep 720, the path visualization application 112 detects the location(s)of one or more control points 410 and user input within the image. Next,at step 730, the path visualization application 112 determines whetherthe user input is modifying a control point. For example, at step 720,the path visualization application 112 could detect the locations of theuser's fingers and, at step 730, determine whether the user hasperformed a gesture (e.g., a grasping gesture, pushing gesture, orpulling gesture) proximate to one or more of the control points 410.

If the path visualization application 112 determines that the user inputis not modifying a control point 410, then the method 700 returns tostep 710. If the path visualization application 112 determines that theuser input is modifying a control point 410, then the method 700proceeds to step 740, where the path visualization application 112modifies the graphical representation of the path 400, the graphicalrepresentation of the robotic device 230, and/or the graphicalindicator(s) 420 based on the modified control point. Then, at step 750,the path visualization application 112 updates the path instructionsbased on the modified control point. The method 700 then returns to step710.

Although the above technique for modifying a path of a component of arobotic device via user input is described with respect to controlpoints 410, the technique could also be performed on path segments 405and/or components of the graphical representation of the robotic device230, such as the end effector 232, joints, arms, etc. For example, atstep 720, the path visualization application 112 could detect user inputproximate to the path segments 405, the end effector 232, joints, arms,etc. Then, at step 730, the path visualization application 112 coulddetermine whether the user input is modifying the locations of the pathsegments 405, end effector 232, joints, arms, etc. At step 740, the pathvisualization application 112 would then modify the graphicalrepresentation of the path 400, the graphical representation of therobotic device 230, and/or the graphical indicator(s) 420 and, at step750, update the path instructions based on the modifications.Additionally, the path visualization application 112 could detect othertypes of user input, such as user input for toggling between differentinverse kinematics states, input for adjusting joint accelerations, etc.

In sum, the path visualization application processes path instructionsto generate a path of one or more components of a robotic device. Thepath visualization application then analyzes an image to detect one ormore reference points within the image. Next, the path visualizationapplication displays a graphical representation of the path of thecomponent of the robotic device, a graphical representation of therobotic device, and/or one or more graphical indicators that provideinformation about the robotic device.

At least one advantage of the techniques described herein is that athree-dimensional representation of the path of a robotic device can beviewed within the operating environment of the robotic device, allowingpotential collisions to be detected, before such collisions take place.Additionally, the path of the robotic device can be modified via userinput within the operating environment itself, and updates to the pathcan be immediately viewed, for example, via an augmented realitydisplay. Accordingly, robotic devices can be more safely and efficientlyimplemented in a variety of processes. Further, detailed informationabout robot components (e.g., joint angles) can be viewed, enabling auser to make more informed decisions when planning complicated movementsof a robot.

The descriptions of the various embodiments have been presented forpurposes of illustration, but are not intended to be exhaustive orlimited to the embodiments disclosed. Many modifications and variationswill be apparent to those of ordinary skill in the art without departingfrom the scope and spirit of the described embodiments.

Aspects of the present embodiments may be embodied as a system, methodor computer program product. Accordingly, aspects of the presentdisclosure may take the form of an entirely hardware embodiment, anentirely software embodiment (including firmware, resident software,micro-code, etc.) or an embodiment combining software and hardwareaspects that may all generally be referred to herein as a “module” or“system.” Furthermore, aspects of the present disclosure may take theform of a computer program product embodied in one or more computerreadable medium(s) having computer readable program code embodiedthereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

Aspects of the present disclosure are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, enable the implementation of the functions/acts specified inthe flowchart and/or block diagram block or blocks. Such processors maybe, without limitation, general purpose processors, special-purposeprocessors, application-specific processors, or field-programmableprocessors or gate arrays.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

While the preceding is directed to embodiments of the presentdisclosure, other and further embodiments of the disclosure may bedevised without departing from the basic scope thereof, and the scopethereof is determined by the claims that follow.

What is claimed is:
 1. One or more non-transitory computer-readablemedia including instructions that, when executed by one or moreprocessors, configure the one or more processors to perform the stepsof: processing path instructions that comprise coded instructions andspecify an operation to be performed by a robotic device; detecting atleast one reference point within a first image of a workspace;generating a representative path associated with the robotic devicebased on the processed path instructions and the at least one referencepoint, wherein generating the representative path comprises: detectingthe at least one reference point within a second image of the workspace;determining that the at least one reference point has moved from a firstlocation to a second location; and modifying the representative pathbased on the second location; and displaying, via a visualizationapplication, the representative path within the workspace.
 2. The one ormore non-transitory computer-readable media of claim 1, whereingenerating the representative path further comprises at least one oftranslating the representative path based on a location of the at leastone reference point, or rotating the representative path based on anorientation of the at least one reference point.
 3. The one or morenon-transitory computer-readable media of claim 1, wherein displayingthe representative path comprises overlaying, via a head-mounted displaydevice, the representative path on a live view of the workspace.
 4. Theone or more non-transitory computer-readable media of claim 1, whereindisplaying the representative path comprises overlaying therepresentative path on substantially real-time images of the workspaceacquired via an image sensor.
 5. The one or more non-transitorycomputer-readable media of claim 1, wherein the representative pathcomprises a path associated with an end effector of the robotic device.6. The one or more non-transitory computer-readable media of claim 1,wherein the representative path comprises an animated three-dimensionalrepresentation of the robotic device.
 7. The one or more non-transitorycomputer-readable media of claim 1, wherein the representative pathcomprises volumetric boundaries occupied by the robotic device whiletraveling along the representative path.
 8. The one or morenon-transitory computer-readable media of claim 1, further comprisingdisplaying, proximate to the representative path, a graphical indicatorthat includes at least one of an angle, a speed, and an accelerationassociated with a joint of the robotic device.
 9. The one or morenon-transitory computer-readable media of claim 1, wherein modifying therepresentative path comprises: modifying at least one of a size, anorientation, and a location of the representative path based on thesecond location.
 10. The one or more non-transitory computer-readablemedia of claim 1, further comprising: detecting user input within thesecond image of the workspace proximate to at least one point on therepresentative path; modifying at least one of the path instructionsbased on the user input; and updating the representative path based onthe at least one path instruction.
 11. A method for displaying arepresentative path associated with a robotic device, the methodcomprising: processing path instructions that comprise codedinstructions and specify an operation to be performed by a roboticdevice; detecting at least one reference point within a first image of aworkspace; generating the representative path based on the processedpath instructions and the at least one reference point, whereingenerating the representative path comprises: detecting the at least onereference point within a second image of the workspace; determining thatthe at least one reference point has moved from a first location to asecond location; and modifying the representative path based on thesecond location; and displaying, via a visualization application, therepresentative path within the workspace.
 12. The method of claim 11,wherein generating the representative path further comprises at leastone of translating the representative path based on a location of the atleast one reference point, or rotating the representative path based onan orientation of the at least one reference point.
 13. The method ofclaim 11, wherein displaying the representative path comprises at leastone of overlaying, via a head-mounted display device, the representativepath on a live view of the workspace and overlaying the representativepath on substantially real-time images of the workspace acquired via animage sensor.
 14. The method of claim 11, wherein the representativepath comprises a path associated with an end effector of the roboticdevice.
 15. The method of claim 11, wherein the representative pathcomprises an animated three-dimensional representation of the roboticdevice.
 16. The method of claim 11, wherein the representative pathcomprises volumetric boundaries occupied by the robotic device whiletraveling along the representative path.
 17. The method of claim 11,further comprising displaying, proximate to the representative path, agraphical indicator that includes at least one of an angle, a speed, andan acceleration associated with a joint of the robotic device.
 18. Themethod of claim 11, wherein modifying the representative path comprises:modifying at least one of a size, an orientation, and a location of therepresentative path based on the second location.
 19. The method ofclaim 11, further comprising: detecting user input within the secondimage of the workspace proximate to at least one point on therepresentative path; modifying at least one of the path instructionsbased on the user input; and updating the representative path based onthe at least one path instruction.
 20. A system for displaying arepresentative path associated with a robotic device, the systemcomprising: a memory storing a path visualization application; aprocessor that is coupled to the memory and, when executing the pathvisualization application, is configured to: process path instructionsthat comprise coded instructions and specify an operation to beperformed by a robotic device; detect at least one reference pointwithin a first image of a workspace; and generate the representativepath based on the processed path instructions and the at least onereference point, wherein generating the representative path comprises:detecting the at least one reference point within a second image of theworkspace; determining that the at least one reference point has movedfrom a first location to a second location; and modifying therepresentative path based on the second location; and a display coupledto the processor and configured to display the representative pathwithin the workspace.
 21. The one or more non-transitorycomputer-readable media of claim 1, wherein generating therepresentative path further comprises scaling the representative pathbased on a size of the at least one reference point.